Multi label vs. Multi class
An expriment in loss functions on the same dataset.
- Tank classifier models:
- Bulding a model using multi class loss
- Bulding a model using multi label loss
- Addtional validation (out of train scope images)
tank_types = 'merkava mk4','M1 Abrams','water'
path = Path('tanks')
#downloading 150 images to labeled directories
if not path.exists():
path.mkdir()
for o in tank_types:
dest = (path/o)
dest.mkdir(exist_ok=True)
urls = search_images_ddg(f'{o} tank', max_images=150)
download_images(dest, urls=urls)
#deleting not working files
fns = get_image_files(path)
failed = verify_images(fns)
failed.map(Path.unlink);
tanks = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parent_label,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls = tanks.dataloaders(path)
dls.valid.show_batch(max_n=5, nrows=1)
learnMC = cnn_learner(dls, resnet18, metrics=error_rate)
learnMC.fine_tune(4) #fastai already chooses multi class loss becuase od the CategoryBlock
Review top 10 loss images:
def parentlabel(x):
return [x.parent.name] # as get_y recieve a list
tanks2 = DataBlock(
blocks=(ImageBlock, MultiCategoryBlock),
#MultiCategoryBlock(add_na=True)
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parentlabel,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls2 = tanks2.dataloaders(path)
dls2.valid.show_batch(nrows=1, ncols=5)
learnML = cnn_learner(dls2, resnet18, metrics=accuracy_multi)
#defaults for the accuracy multi are: threshold=0.5, Sigmod=True
learnML.fine_tune(4)
Addjusting the threshhold (as the curve is smooth we shuold nit be worried about overfitting):
preds, targs = learnML.get_preds()
xs = torch.linspace(0.05,0.95,29)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);
learnML.metrics = partial(accuracy_multi, thresh=0.85)
learnML.validate()
Review top 10 loss images:
To use our model in an application, we can simply treat the predict method as a regular function.
The first model is binary and forces to select a single class based on any image input. Its softmax encourages a single class-selection even more. While the second model is multi-label and predicts each class separately, it may be better at alerting images outside the scope of the class or two classes within one image.
Now let's see if one model can perform better on out-of-scope images even though they were both trained on the same dataset:
multi_learner2.predict('/content/DUDU.jpg')
multi_learner2.predict('/content/TetroWtank.jpg')
multi_learner.predict('/content/manytanks.jpg')
multi_learner.predict('/content/merk_abrms2.jpeg')
multi_learner.predict('/content/merkav_abrams.jpg')
multi_learner.predict('/content/mek_abs_3.jpg')
#img.show()
#learn.predict(img)[0]
multi_learner.recorder.plot_loss()
multi_learner.show_results()
learn.recorder.plot_loss()
x, y = dls.one_batch()
x.shape
y
learn.metrics[0]
learn.loss_func